test: property-check model-map resolution invariants#597
Conversation
Six fast-check properties over the model resolution pipeline (resolveNormalizedModel / getModelProfile / getNormalizedModel), generating known aliases, synthesized GPT-5 spellings (minors 0-9, mini/nano/pro/codex variants, -/./space separators, suffixes), provider prefixes, random casing, and raw garbage: - closed world: every resolution lands on a MODEL_PROFILES key, so getModelProfile's DEFAULT_MODEL fallback is pure defence - idempotence: normalized outputs are fixpoints - provider prefixes and casing never change the resolution - codex dominance: unmapped ids mentioning codex resolve to CURRENT_CODEX_MODEL, never a general model - the inverse: unmapped general GPT-5 spellings stay codex-free and in the gpt-5 family (no silent codex routing for general requests) - every explicit MODEL_MAP alias resolves to its mapped target under any prefix/casing spelling Companion to the property suites in #574/#575/#592-#596. https://claude.ai/code/session_01XNtnkLbBiXZxfQQYLMpucB
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Warning Review limit reached
More reviews will be available in 14 minutes and 14 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more credits in the billing tab to continue. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary
lib/request/helpers/model-map.ts— the parser every request's model routing flows through (resolveNormalizedModel→getModelProfile). The test: property-check the unsupported-model fallback invariants #574 suite property-checked the unsupported-model fallback chain; this one pins the resolution layer beneath it.What Changed
New
test/property/model-map.property.test.ts(6 properties). Inputs mix knownMODEL_MAPaliases, synthesized GPT-5 spellings (minors 0–9, mini/nano/pro/codex variants,-/./space separators, version suffixes), provider prefixes (openai/,models/, nested), random per-character casing, and raw garbage strings:MODEL_PROFILESkey, provinggetModelProfile'sDEFAULT_MODELfallback is pure defence (no reachable input needs it; aMODEL_MAPvalue or catalog entry without a profile would be found instantly).codexresolve toCURRENT_CODEX_MODEL, never a general model.gpt-5family: a general-purpose request can never silently route to a codex-tuned model.MODEL_MAPalias resolves to its mapped target under any prefix/casing spelling (sweeps the whole map every iteration, so a future alias edit that breaks case-insensitive lookup fails immediately).No SUT bugs found.
Validation
npm test -- test/property/model-map.property.test.ts test/model-map.test.ts— 21/21 (new 6 + existing 15 untouched)npm run typecheck(also via pre-commit hook)npx eslint test/property/model-map.property.test.ts --max-warnings=0Docs and Governance Checklist
Risk and Rollback
test/property/suites.https://claude.ai/code/session_01XNtnkLbBiXZxfQQYLMpucB
Generated by Claude Code
note: greptile review for oc-chatgpt-multi-auth. cite files like
lib/foo.ts:123. confirm regression tests + windows concurrency/token redaction coverage.Greptile Summary
adds
test/property/model-map.property.test.ts— a six-property fast-check suite that pins the resolution layer ofresolveNormalizedModel/getModelProfile(the path every request's model routing flows through). no sut or production code is changed.MODEL_PROFILESkey, and normalized outputs are fixpoints of the resolver.stripProviderPrefix+ case-fold runs safely as one path for synthesized, mapped, and garbage inputs.CURRENT_CODEX_MODEL; general GPT-5 spellings never silently route to a codex-tuned model; everyMODEL_MAPalias resolves to its declared target under any prefix or casing.Confidence Score: 5/5
test-only addition with no production code changes; safe to merge
single additive test file; all six properties are logically sound against the sut — every resolveNormalizedModel exit path returns a MODEL_PROFILES key, all catalog outputs are self-aliases in MODEL_MAP ensuring idempotence holds, and the combined prefix+casing assertion in property 3 closes the gap flagged in the previous review thread
no files require special attention
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD IN["arbModelId\n(MAP_KEY | synthesized GPT-5 | garbage)\n+ arbPrefix"] --> RNM["resolveNormalizedModel"] RNM --> SPP["stripProviderPrefix\n(take last '/' segment)"] SPP --> LMM{"lookupMappedModel\n(exact, then case-insensitive)"} LMM -->|hit| OUT1["MODEL_MAP value"] LMM -->|miss| RCC{"resolveCodexCatalogModel\nincludes('codex')?"} RCC -->|yes| OUT2["CURRENT_CODEX_MODEL"] RCC -->|no| RGC{"resolveGeneralGpt5CatalogModel\ntokenize to GPT-5 family?"} RGC -->|yes| OUT3["catalog entry"] RGC -->|no| OUT4["DEFAULT_MODEL (gpt-5.5)"]Reviews (2): Last reviewed commit: "test: combine prefix and casing mutation..." | Re-trigger Greptile